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I. INTRODUCTION 



1. Background of the Study 

To assist higher education, the Congress of the United States 
enacted the Higher Education Facilities Act of 1963. Title 1 of this Act 
made Federal funds available to public and private institutions to build 
urgently needed academic facilities and designated the Board of Regents 
of The University of the State of New York as the State Commission to 
administer the program in New York State. 

The primary responsibility of the State Commission is to process 
applications from institutions and to determine relative priorities for 
proposed construction projects through standardised criteria and pro- 
cedures. To do this, the State Commission adopted a State Plan for the 
facilities program* which stipulated the administrative rules for the 
FederaL grants program the State level. Using criteria and procedures 
prescribed in the State Plan, the Commission recommended grants totaling 
approximately $107 million to seventy-eight 4-year institutions during the 
past 5 years. 

In the beginning of the program, a number of questions *ere raised 
concerning the application evaluation process. A first question pertained 
to the appropriateness of evaluative criteria. Although criteria set by 
the State Commission reflected the guidelines established in the Act, 2 



State Plan for the Higher E ducation Faci 1 i ties Act of 1963 , 
Albany: Office of Planning in Higher Education. State Education 

Department, 1966. See also the amended State Plan which was approved in 
July 1968. 

2 In the Higher Education Facilities Act of 1963, it is stated that 
grants were to be made, "To as?<8t the Nation’s institutions of higher 
education to construct needed classrooms, laboratories, and libraries in 
order to accommodate mojnting student enrollments and to meet demands for 
skilled technicians. :t 
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only a limited number of specific institutional factors were included, 
and those factors were quantitative in nature. The question then, was 
what additional factors, either qualitative or quantitative, should be 
used tc set priorities for project applications? 

A second question concerned the evaluation procedure itself, 

The staff assigned to administer the Title I program cross-checked 
application data against other sources. Each institution received points 
for each criterion according to predetermined scales. The points accrued 
for all criteria and the priority of each proposed project was finally 
determined by numerical rank. This procedure insured a very high degree 
of objectivity in the evaluation of applications and enabled the State 
Commission to respond to any question or complaint about the results of 
its evaluation procedure, The accountability of the Coinmission's recom- 
mendations to the Office of Education in Washington was also enhanced, 

However, to some administrators, the evaluation procedure seemed 
mechanical and statistically questionable. An institution of higher 
education is so complex and dynamic that quantitative evaluation may be 
insufficient. Fci example, identical totals from different scores on the 
various criteria may not accurately represent the same degree of space 
needs of institutions, Thus, the need for another way to evaluate was rec- 
ognized, One suggestion was the use of a panel of impartial judges to rate 
i ns ti tutions . 

A third question related to the value of university personnel in 
the administration of the Grants Program. The State Commission had already 
appointed a special advisory committee on higher education academic 
facilities planning composed of higher education officials and lay leaders 
drawn from various fields in the State. This committee and a standing 



advisory committee on higher education decided that the Commission could 



directly utilize the knowledge and experience of leaders in institutions of 
higher education for the evaluation of grants applications. 

Lastly, a suggestion was made that the final decision on the distri- 
bution of facilities grants funds be based not only on the space needs of 
institutions, but also on their educational cc tribution to New York State. 
Whether this latter criterion is appropriate is largely judgmental; however, 
the educational contribution of institutions, and its relationship to space 
needs can be empirically studied. 

Recognizing the importance of the foregoing questions, the Bureau 
of Higher Education Facilities Planning in the Office of Planning in 
Higher Education, which administers the Grants Program, requested an 
empirical study of the problem by the Bureau of Research in Higher and 
Frofessional Education. 

2. Purpose and Significance of the Study 

The primary purpose of this study is to examine the question of 
criteria used for evaluating applications for Grants made under Title I of 
the Higher Education Facilities Act of 1963 and to investigate the appropri- 
ateness of using a panel of judges, composed primarily of college adminis- 
trators, in application evaluation. The judges' criteria evolved through 
the analysis of factors considered in rating institutions. The usefulness 
of the judges' ratings is determined by examining the validity, reliability, 
and objectivity of these ratings. 

Ihis study does not deal directly with the evaluative procedures 
presently used by the State Commission, but it does compare the Corranission' s 
criteria with those developed by a panel of judges. However, the focus of 
this study is on the analysis of the rating behavior of judges. 
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Although this study developed from the New York State Commissions 
responsibility in the distribution of Higher Education Facilities grants, 
the practical significance of the study may be extended beyond that 
responsibility. In recent years, both the Federal and State Governments 
have provided increasingly greater sums of money to colleges and univer- 
sities in the United States, and predictions are that this roLe of 
government in funding higher education will continue, as shown in table 1. 

Currently, an important issue In higher education is the concept of 
the administration of governmental aid programs. Specific questions are: 
(1) Should Federal aid be granted to states, institutions, or individuals 
in the institutions? (2) How shouLd personnel administering government 

3 

aid programs be organized at the different governmental levels? (3) How 
should decisions on the distribution of the grants be made? (4) How can 
the impact of governmental aid on the educational community be assessed?^ 
These are only a few of the questions being asked about governmental aid. 
As the demand for governmental aid increases and consequent implications 
for higher education become more profound, empirical knowledge in this 
area is needed. This study may pr<vide such knowledge, particularly in 
regard to the third question, i.e., how should decisions on the distribu- 
tion of grants be made? 



3 

This and the preceding question are critically discussed in Douglas 
M, Knight, et al. The Federal Government and Higher Education , Englewood 
Cliff s, N. J . : Prentice-Hall, 196C, pp. 165-175; Alice M. Rivlin. The Ro le 

of the Federal Government in Financing Higher Education , Washington, D.C.: 
The Brookings Institute, 1961, pp. 156-175. 

4 

Clark Kerr, The Uses of the University , Cambridge: Harvard 

University Press, 1963, pp. 46-84. 
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Table 1 



Sources of Funds for Higher Education* 



Mmounts in Billions of Dollars/ 



Sources of Funds 


1959-60 


1968-69 


1980-81 

(Projected) 


Amount 


Percent 


Amount | 


Percent 


Amount 


Percent 


Student Tuition and Fees 


1.2 


17 


3.6 


18 


7.0 


18 


Federal Government 


1.0 


14 


4.8 


24 


10.9 


2-3 


State and Local Government 


1.5 


21 


5.2 


25 


8.6 


22 


Gifts and Endowment Earnings 


0.6 


8 


2. 1 


10 


3.4 


9 


Income of Auxiliary 
Enterprises 


1. 1 


16 


2.4 


12 


3.5 


9 


Other 


1.7 


24 


2.3 


11 


5.6 


14 


Total 


7.1 


100 


20.4 


100 


39.0 


LOO 



* Adopted and adjusted from the following two souiees: Robert A. 

Freeman. Crisis in Co 1 lege Finance , Washington, D.C.: The Institute for 

Social Science Research, 1965, p. 186; and Howard R. Bowen’s estimations 
reported in The Chronicle of Higher Education , November 17, 1969, p. 4. 

In a theoretical framework, it is possible to study the decision- 
making process using analytical techniques for assessing institutional 
evaluation criteria, with no anchor to reality. Unfortunately, empirical 
studies of the decisionmaking process in existing institutions of higher 
education are very rare. The present study is an attempt to fill this 
void. 
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11. THE PROBLEM 



Thu purpose of this study is to examine the usefulness of a panel 
of judges in ranking the applications for facilities grjnts submitted ‘by 
institutions of higher education. The three major aspects of the judges' 
ratings to be analyzed are the validity, reliability, and objectivity of 
the evaluation method. The examination of the judges' criteria for the 
evaluation of grant applications is dealt with in the section on validity 
analysis . 

1 . Val idi ty 

In psychometrics, the validity of a measurement tool or technique 
is the degree to which it measures what it is supposed to measure. This 
concept involves the identification of factors that account for any 
variation in measured values, With respect to such identification, the 
problem to bo investigated is: What characteristics of institutions of 

higher education, as set forth in the study, influence the responses of 
impartial judges? The judges were asked to rate the space needs and the 
educational contribut ion of individual institutions. The specific ques- 
tions relating to validity are: 

A. What institutional factors are related to the 
judges* ratings of the spece needs and the 
educational contribution of institutions of 
higher education? 

B. What is the relationship between the scores 
achieved using judges* ratings and scores 
obtained through the rating procedures used 
by the State Commission? 

13 
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C. What differences may be observed in the responses of 
judges as they rate institutional space needs and 
educational contributions? 

D. To what extent are the judges' ratings actually re- 
lated to identified institutional factors which they 
consider important? 

2. Reliabilit y 

Reliability means the degree of consistency of the rating of the 
same object or institution made by individuals, or the degree of 
consistency of the ratings made by the same judges over a period of time. 

In this study, the first definition is used, i.e., the consistency of ratings 
made by a number of judges, which may also be called "inter judge consistency." 
As an additional measure, the variability of scores given to individual in- 
stitutions is used as an indirect indicator of reliability. Specific 
questions relating to reliability are: 

A, What is the variance of scores given by judges 
to individual institutions? 

B. What is the internal consistency^ of the judges' 
ratings? 

3 . Objectivity 

The last aspect of the problem is the identification of any 
personal bias involved in ratings. Three factors which may hypothetically 
reduce the objectivity of ratings are indicated by these questions: 

^Determined by the Kruskal-Wa 1 lis one-way analysis of variance among 
subgroups of judges. 







A, 



How does a judge's affiliation with a particular 
institution affect his rating of that institution? 

B. How does the type of control of the institution 
with which a judge is affiliated affect his rating 
of an institution under the same type of control? 

C. How does the geographical area of the institution 
with which a judge is affiliated affecl his rating 
of an institution in the same area? 

The fundamenta 1 emphasis of this study is investigative, not 
determinative. Therefore, Lhe study does not intend to formulate 
hypotheses corresponding to specific questions prior to implementation 
of the empirical study. 

Another facet of this study is methodological. Emphasis is placed 
on an analysis of the feasibility of a particular evaluation technique, i.e. 
rating by judges, in setting priorities under which institutions would 
receive Federal grants. Therefore, more attention is paid to the 
perceptions of judges than *:o standard determinants of the space needs or 
the educational contribution of institutions. 

111. METHODS AND PROCEDURES 

As an alternate method to the assessment of an institution's space 
needs based on quantitative criteria as established in the State Plan for 
the administration of the Higher Education Facilities Act of 1963, a panel 
of 34 judges was asked to rate 60 institutions based on their evaluative 
judgments of (1) actual space needs, and (2) the educational contributions 
of the institutions. Data sheets containing various institutional 
characteristics were provided to assist thr judges in their ratings. Sort 




ordering of the institutions from 1 to 60 was recorded. The judges' 
ratings were then statis ticaL Ly analyzed. 

1 . Selection of Judges 

Thirty-seven administrators of colleges and universities with 
wide experience in higher education in New York State were initially 
selected from both public and private 2- and 4-year institutions. In 
accordance with the study plan, 30 persons of the initial 37 were selected 
to form the panel of judges; among them nine were presidents, 10 were vice 
presidents, five were deans of students, and six were institutional re- 
searchers. To this group were added four persons from the New York State 
Education Department, i.e., individuals included for comparison purposes. 
The selection of the 34 judges was in a measure arbitrary, restricted by 
the small number of judges needed and by the need to ascertain beforehand 
a willingness to serve. The judges chosen, therefore, were not necessarily 
representative of college and university administrators in New York State. 

2. Characteristics of Institutions 

The 60 institutions in this study were 4-year colleges and 
universities which applied for Federal facilities giants ir the period 
from 1964 to 1967. ^ Among them, 46 are private and 14 are public 
institutions. They included, in almost equal numbers, metropolitan, urban, 
and rural ins ti tuticns. Eleven have less than 1,000 full-time eqi ivalent 



Only 4-year institutions are included in this study, although 
Title I of the Higher Education Facilities Act of 1963 covers public 
community colleges and public technical institutions as well as 4-year 
ins ti tutions . 
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(FTE)^ students, 21 have ^ ,ween 1,000 and 1,999 FTE students, 21 have 
2,000 to 9,999 FTE students, and seven have 10,000 or more FTE students. 



The 60 institutions were not randomly selected from all the 



institutions in New York State, but were se lf-se lec ted in the sense that 
they applied for the Federal facilities grant funds. However, the institu- 
tions represented an adequate range of student enrollment, institutional 
control, and geographical areas. 

3. Institutional Data Sheet 

Each judge was provided with 60 data sheets corresponding to the 
60 institutions of higher education included in the study. Each sheet 
contained information related to the eight factors used by the State 
Commission for application evaluation, and data on 23 additional factors 
(table 2, and Appendix I). In the study, separate items of data are 
treated as independent variables; however, it cannot be assumed that they 
are the only factors which determined the ratings of the judges. 



7 FTE is defined as all full-time students and one-third of patt- 
titne students. 




- 10 - 



Table 2 



Institutional Factors Indicated on Data Sheet 



I. STATE COMMISSION FACTORS 

1. Enrollment increase (percentage) 

2. Enrollment increase (numerical) 

3. Increase in instructional and library 

space (percentage) 

4. Increase in instructional and library 

space (numerical) 

5. Utilization of facilities 

6. Date of most recent grant 

7. Amount of previous grant 

8. Evidence of a long-range plan 

II. ADDITIONAL FACTORS 

9. Control of institution 

10. Type of community 

11. Academic calendar 

12. Endowment 

13. Number of full-tima students 

14. Number of part-time students 

15. Number of residential students 

16. Number of nonresidential students 

17. Number of in-State students 

18. Number of out-of-State students 

19. Number of faculty members 

20. Faculty/student ratio 

21. Number of faculty with doctorate 

22. Average faculty salary 

23. Provision for an honors program 

24. Number of undergraduate degrees 

granted 

25. Number of masters degrees granted 

26. Number of doctorates granted 

27. Number of volumes in the library 

28. Number of volumes acquired yearly 

29. Ratio of volume s/student 

30. Type of institution 

31. Scope of curriculum offerings 
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4. Procedures for Rating 



The judges were asked to rate each institution twice on the basis 
of the information provided on the data sheets: (l) the extent to which 

the institution was fulfilling the congressional mandate, and (2) the 
extent to which the institution was contributing to the quality of higher 
education in New York State (Appendix II). 

An instruction sheet, 60 data sheets, and an evaluation recording 
sheet were mailed to each judge. The instruction sheet set the rating 
procedures as follows: 

Evaluation 1: 

(1) Sort the 60 institutions into seven groups according 
to your judgment of the institution’s space needs 
based on data supplied. 

(2) After the sorting is completed, indicate the code 
number of each institution on the Recording Sheet 
for Evaluation from l (greatest need) to 60 (least 
need) . 

Evaluation 2: 

(1) Sort the 60 institutions into seven groups accord- 
ing to your evaluation of the institution's 
contribution to higher education in the State. 

(2) Record each institutions code number on the 
Recording Sheet as in (2) above. 

For the sake of statistical analysis, the ordinal measure of 
institutional rankings was transformed into an interval measure based on 
the concept of normal distribution. As Appendix 111 shows, the Recording 
Sheet for Evaluation was designed to indicate the converted score of a 

19 
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group of rankings-- 1 point for lst-2nd, 2 for 3rd-9th, 3 for 10th-22nd, 

4 for 23rd-38th, 5 for 39th-51st, 6 for 52nd-58th, 7 for 59th-60th. This 
transformation was made to insure a more valid arithmetical calculation in 
data analysis. In this scale, a low score indicated a high degree of 
space need or educational contribution, and a high score indicated a low 
degree of need, However, in presenting the data, a higher score earned 
indicates a higher degree of space needs or educational contribution. 

5 . Methods of Statistical Analyse^ 

Methods of statistical analyses used were single and multiple 
regression analysis and chi square techniques, whichever was most appropriate 
to the specific problem. Another technique used was the Kruskal -Wa llis one- 
way analysis of variance. 

IV. PRESENTATION AND ANALYSIS OF DATA 
1 , Validity of Ratings 

In this section, the validity of the ratings by 34 judges was 
examined by an analysis of the data collected. The major focus of this 
section was: (1) to identify the institutional characteristics which 

related to the judges' ratings, and (2) to examine the appropriate criteria 
for the evaluation of facilities grants. 

A Factors Relating to Judges 1 Ratings 

Each judge was asked to rate on a seven point scale: (1) the 

space needs of each institution, and (2) the quality of the educational 
contribution made by that institution. The judges were given data sheets 
listing items of institutional data, and were asked to rate on that basis, 
and also on their own knowledge and Judgment of the institution, Therefore, 
it cannot be assumed that the judges 1 ratings were determined exclusively 
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